Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus that is able to prevent negative influence caused by an illuminating unit of an external device emitting light or flashing when specifying the external device, wherein the image processing apparatus includes a control unit configured to control a display screen that performs superimposed display of information relating to an external device on a video image including the external device that has been captured by an image capturing unit; and determining unit configured to determine whether the illuminating unit of the external device is in use or not, or whether an image recognition operation based on the video image is in progress or not, wherein when an instruction to make an illuminating unit of the external device light up or flash has been input, in a case in which determining unit determines that the illuminating unit of the external device is in use, or an image recognition operation based on the video image is in progress, the control unit displays a screen for displaying predetermined specifying information in the vicinity of the external device on the display screen instead of making the illuminating unit light up or flash.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, a storage medium, and the like.

Conventionally, image processing apparatuses such as monitoringapparatuses and the like are known that are able monitor video imagesand acquire audio by control through remote operations via a network ora leased line. In these monitoring apparatuses, it is possible toacquire and control information by connection with an external device.In addition, the external device is generally provided with an LED(light emitting diode) that serves as an indicator, and for example,specifying the external device, or the like is performed by lighting upthe indicator.

In Japanese Patent No. 6055421, a display apparatus is disclosed inwhich, in a case in which the display apparatus has itself been selectedas the communication partner from the side of a different image displayapparatus, the display apparatus makes an indicator provided in thevicinity of the device's own image display unit light up at apredetermined brightness.

In Japanese Unexamined Patent Application, First Publication No.2002-354363, when the power for a video apparatus is turned on, a systemprovided with a user interface that selectively lights up theillumination of its power indicator is opened.

In contrast, in a case in which an electric light type external deviceis controlled via a network or the like, there are cases in which theelectric light itself is also given the role of serving as theindicator. In such cases, it is possible that upon instructing theindicator to light up or flash, there is a concern that the electriclight lighting up or flashing will affect the surrounding lightingenvironment.

The present invention was created taking into consideration problemssuch as that described above, and one of its goals is to provide animage processing apparatus, or the like in which it is possible toprevent the negative effects caused by the illuminating unit of anexternal device emitting light or flashing when specifying the externaldevice.

SUMMARY OF THE INVENTION

An image processing apparatus according to one aspect of the presentinvention comprises at least one processor or circuit configured tofunction as: a control unit configured to control a display screen thatdisplays a superimposed display of information relating to an externaldevice on a video image including the external device that has beencaptured by an image capturing unit, and determining unit configured todetermine whether the illuminating unit of the external device is in useor not, or whether an image recognition operation based on the videoimage is in progress or not, wherein when an instruction to make anilluminating unit of the external device light up or flash has beeninput, in a case in which determining unit determines that theilluminating unit of the external device is in use, or an imagerecognition operation based on the video image is in progress, thecontrol unit displays a screen for displaying predetermined specifyinginformation in the vicinity of the external device on the display screeninstead of making the illuminating unit light up or flash.

Further features of the present invention will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing one example of a configuration of amonitoring system 10 in a First Embodiment.

FIG. 2A is a functional block diagram showing one example of aconfiguration of a monitoring apparatus 1 according to the FirstEmbodiment. FIG. 2B is a functional block diagram showing one example ofa configuration of a client apparatus 2 according to the FirstEmbodiment.

FIG. 3 is a diagram showing one example of a display screen 31 forexternal device management settings in a monitoring apparatus 1according to the First Embodiment.

FIG. 4 is a diagram showing one example of a display screen 41 formaking a superimposed setting of a “current value” of an external devicein a monitoring apparatus according to the First Embodiment.

FIG. 5 is a diagram showing one example of a display screen 51displaying a superimposed image according to the First Embodiment.

FIG. 6 is a flowchart showing one example of processing for a monitoringapparatus according to the First Embodiment.

FIG. 7 is a diagram showing one example of a display screen promptingdisplay of specifying information on a display unit of a clientapparatus 2 according to the First Embodiment.

FIG. 8 is a diagram showing one example of a display screen on whichspecifying information has been superimposed on a viewer according tothe First Embodiment.

FIG. 9 is a diagram showing one example of a display screen 91 forsetting an image detection for an external device in a monitoringapparatus 1 according to a Second Embodiment.

FIG. 10 is a flowchart showing one example of processing for amonitoring apparatus according to the Second Embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the accompanying drawings, favorablemodes of the present invention will be described using Embodiments. Ineach diagram, the same reference signs are applied to the same membersor elements, and duplicate descriptions will be omitted or simplified.

First Embodiment

FIG. 1 is a diagram showing one example of a configuration of amonitoring system 10 according to a First Embodiment. The monitoringsystem 10 of the present embodiment is provided with a monitoringapparatus 1, a client apparatus 2, external devices 3 a, and 3 b, andthe like. In the present embodiment, for example, the monitoringapparatus 1, and the client apparatus 2 function as image processingapparatuses.

The monitoring apparatus 1 is a monitoring camera that captures imagesof subjects such as people or the like in video images, and, forexample, is able to use a network camera. The monitoring apparatus 1performs communications with the client apparatus 2, and the externaldevices 3 a, and 3 b via networks 4, and 5.

The client apparatus 2 is an apparatus for controlling the operations ofthe monitoring apparatus 1, and, for example, is able to use aninformation processing device such as a PC (personal computer), or thelike. Note that the client apparatus 2 has a display unit 6, andfunctions as a display control apparatus that controls the display unit6. In addition, the client apparatus 2 transmits each type of controlcommand to the monitoring apparatus 1.

Control commands include, for example, operation (light up, and thelike) instructions for the external devices 3 a, and 3 b, and the like,light up or flash instructions for the indicator in order to specify(confirm) the external devices 3 a, and 3 b, commands for setting imagedetection, performing control, and the like of the monitoring apparatus1, and the like. Note that the monitoring apparatus 1 transmitsresponses to the control commands that have been received to the clientapparatus 2.

The external devices 3 a, and 3 b perform communications with themonitoring apparatus 1 via the network 5. The monitoring apparatus 1transmits control commands to the external devices 3 a, and 3 b, andreceives notification commands from the external devices 3 a, and 3 brelating to the status of the devices.

In the present embodiment, the external devices 3 a, and 3 b areelectric light type devices, and are configured so as to make theindicator provided on the housing of the external device light up orflash by operating an electronic circuit in response to externallyreceived control commands.

Note that in the present embodiment, the indicator may be anilluminating unit such as an LED or the like, or the electric lightitself may also be used as the indicator. In this context, the indicatorand electric light function as an illuminating unit.

In addition, the external devices 3 a, and 3 b transmit notificationcommands to the monitoring apparatus 1 that notify the monitoringapparatus 1 of their current state (for example, current values such asthe brightness of the electric light, or the like). Note that thefunctions held by the external devices 3 a, and 3 b are not limited toelectric light functions, and for example, the external devices 3 a, and3 b may also be devices having other functions such as various types ofsensor device.

The monitoring apparatus 1 and the client apparatus 2 are connected in astate in which they can communicate with each other via the network 4.The network 4 is configured by, for example, a plurality of routers,switches, cables, and the like that fulfill a communication standardsuch as Ethernet (registered trademark), or the like. However, anycommunication standard, scale, and confirmation may also be used ifcommunication between the monitoring apparatus 1 and the clientapparatus 2 is possible.

The monitoring apparatus 1 and the external devices 3 a, and 3 b areconnected in a state in which they are able to communicate with eachother via the wireless or wired network 5. The network 5 can use, forexample, a network according to a communication standard such as Z-Wave,Zigbee (registered trademark), wireless LAN, or the like. However, anycommunication standard, scale, or configuration can be used for thenetwork 5 if communication is possible between the monitoring apparatus1 and the external devices 3 a, and 3 b.

Upon receiving a light up or flash instruction command for the indicatorfrom the monitoring apparatus 1, the external devices 3 a, and 3 b makethe indicator (electric light, LED, or the like) light up or flash. Onepurpose of making the indicator light up or flash is specifying(confirming) which device among the external devices that are installedis the external device that is operating on the display screen. However,the purpose of the indicator is not limited thereto, and for example, itmay also be used in order to check the operating status of the externaldevices 3 a, and 3 b.

Note that although in the present embodiment, an example is explainedbelow in which the indicator is made to flash, the indicator may also bemade to light up. In a case in which the communication standard for thenetwork 5 is Z-Wave, the indicator flash instruction command uses anindicator set command, or the like.

The flashing format of the indicator such as the strength, flashingtime, flashing number, and the ratio of light emitting and non-lightemitting times during one flash during the light is being emitted forthe indicator flashes may be configured to be set by indicator flashinstruction commands, or may also flash in a flashing format that hasbeen determined in advance by the external device.

Next, FIG. 2A is a functional block diagram showing one example of aconfiguration of the monitoring apparatus 1 according to the FirstEmbodiment, and FIG. 2B is a functional block diagram showing oneexample of a configuration of the client apparatus 2 according to theFirst Embodiment.

Note that a portion of the functional blocks shown in FIG. 2A, and FIG.2B are realized by a computer that is included in the monitoringapparatus 1 or the client apparatus 2 executing a computer program thathas been stored on a storage unit that serves as a storage medium.However, a portion or the entirety thereof may also be realized byhardware.

An Application-Specific Integrated Circuit (ASIC) and a processor (areconfigurable processor, a DSP), or the like can be used as thehardware. In addition, each of the functional blocks that are shown inFIG. 2A, and FIG. 2B may be encased in the same housing, or they mayalso be configured by separate apparatuses that are connected to eachother via a signal path.

As is shown in FIG. 2A, the monitoring apparatus 1 is provided with acontrol unit 11, a storage unit 12, an image capturing unit 13, acompressing encoding unit 14, a communications unit 15, an imagecapturing control unit 16, and the like. Each configurational unit ofmonitoring apparatus 1 is connected via a bus 17.

As has been described above, a CPU that serves as a computer is providedin the control unit 11, and controls the operations of each unit of theentire monitoring apparatus 1 based on the computer program that hasbeen stored on the memory serving as the storage unit. Note that thecontrol unit 11 controls a display screen that performs superimposeddisplay of information relating to an external device on a video imageincluding the external device that has been captured by the imagecapturing unit 13.

The storage unit 12 stores each type of data. The storage unit 12 mainlyhas a storage area for the computer program that the CPU of the controlunit 11 executes, and a work area for when the program is beingexecuted. In addition, the storage unit 12 is used as a storage area forvarious data such as a storage area for management information for theexternal device, setting information for image detection and the like,setting information for event triggers and action rules, and image datagenerated by the image capturing unit 13 and the control unit 11, andthe like.

In addition, when the monitoring apparatus 1 adds an external device tothe management targets, each type of information for the externaldevice, for example, supporting command information, the device type forthe external device, and the like is acquired and stored on the storageunit 12.

The image capturing unit 13 acquires an analogue signal by capturingimages of a subject image that has been formed by an image capturingoptical system, which is not illustrated. Furthermore, the analoguesignal is converted to a digital data, necessary image processing isperformed, and this is stored on order on the storage unit 12 ascaptured images. The control unit 11 receives image acquisition eventsfrom the image capturing unit 13 in a case in which a captured image hasbeen stored on the storage unit 12.

The compressing encoding unit 14 generates image data by performingcompression encoding processing based on a format such as JPEG (JointPhotographic Expert Group) or H.264, and the like on the captured imageor generated superimposed image that has been stored by the imagecapturing unit 13, and stores these on the storage unit 12.

The communications unit 15 receives control commands from externalapparatuses such as the client apparatus 2 and the like, and transmitsresponses to the control commands to the external apparatus. In a casein which the communications unit 15 has received a command from anexternal apparatus, the control unit 11 receives a command receptionevent from the communications unit 15.

In addition, the communications unit 15 transmits control commandsgenerated by the control unit 11 to the external apparatus. Furthermore,the communications unit 15 reads out image data that has been compressedand encoded by the compressing encoding unit 14 from the storage unit 12in order and transmits this to the client apparatus 2 to serve as avideo image stream.

The image capturing control unit 16 changes the angle of view and thelike of the image capturing unit 13 by having a drive unit perform eachof pan drive, tilt drive, zoom drive, rotation drive, or focus drivebased on a pan, tilt, rotation, zoom, or focus value that has beeninstructed by the control unit 11. Note that the monitoring apparatus 1may also be a configuration that does not include a portion or theentirety of the plurality of drive units described above. In addition,electronic pan, tilt, rotation, zoom, and the like may also be executedby image extraction.

In addition, as is shown in FIG. 2B, the client apparatus 2 is providedwith a control unit 21, a storage unit 22, an operating unit 24, acommunications unit 25, a display control unit 26, and the like. Eachconfigurational unit of the client apparatus 2 is connected via a bus27.

The control unit 21 is provided internally with a CPU that serves as acomputer, and controls the operations of each unit of the entirety ofthe client apparatus 2 based on a computer program that has been storedon a memory serving as a storage medium.

The storage unit 22 has the same configuration as the storage unit 12,the communications unit 25 has the same configuration as thecommunications unit 15, and the operating unit 24 includes a keyboard, amouse, and a touch panel. 26 is a display control unit, and performscontrol of images displayed on the display unit 6. 27 is a bus.

Next, an example of a display screen displayed on the display unit 6 ofthe client apparatus 2 will be explained. FIG. 3 is a diagram showingone example of a display screen 31 for external device managementsettings in the monitoring apparatus 1 according to the FirstEmbodiment.

The display screen 31 is displayed by the CPU that serves as thecomputer inside of the control unit 21 of the client apparatus 2executing a program that is stored on the storage unit 22 of the clientapparatus 2 based on instructions from a user to perform external devicemanagement settings.

The display screen 31 displays management information 311, and 312 thatdisplay management information for each of the external devices 3 a, and3 b, an add button 313 that adds a new external device, and a deletebutton 314 that deletes a registered external device.

The monitoring apparatus 1 is transitioned to a state in which anexternal device can be added by the user pressing the add button 313.Along with this, by transitioning the external device that the userwould like to add to a state in which it can be added by pressing aphysical button or the like that is provided on the housing of theexternal device, this external device is registered as an externaldevice that the monitoring apparatus 1 manages.

Note that the method for registering external devices is not limitedthereto, and for example, external devices may also be registered usingother methods such as automatically registering external devices basedon identification information for external devices that has beenregistered in the monitoring apparatus 1 in advance, or the like. Inthis manner, the external devices 3 a, and 3 b are registered in themonitoring apparatus in advance, and the management information 311, and312 are displayed.

The management information 311 displays a “device number” foridentifying registered devices, a “type” that shows what kind offunctions are primarily held by the external device 3 a, and a “drivestate” that shows whether or not communication with the external device3 a is in an on state. Furthermore, a “current value” that shows thecurrent state of the functions held by the external device (for example,the current operating rate, or the like) is displayed. This informationis transmitted by the external device 3 a, received by thecommunications unit 15, and stored on the storage unit 12.

In the management information 311 in FIG. 3 , “number 2” is shown in“device number”, “electric light device”, which shows an electric lighttype device, is shown in “type”, “on”, which shows that the device isoperating, is shown in “drive state”, and “80%”, which is the operatingrate of the light emission brightness is shown in “current value”.However, the information that is displayed in the management information311 is not limited to the information described above, and the type andcombination thereof do not matter as long as this is informationrelating to the external device 3 a.

In addition, in “type”, instead of an electric light device,“multi-level switch”, which indicates a device in which numerical valuescan be set, or the like can also be displayed. In addition, a buttonreading “indicator flash”, which makes the indicator of the externaldevice 3 a flash in order to specify the external device 3 a isdisplayed in the management information 311.

In the management information 312 that displays information for theexternal device 3 b, “number 3” is displayed as the “device number”, andthe other information is the same as that in relation to the managementinformation 311.

FIG. 4 is a diagram showing one example of a display screen 41 formaking a superimposed setting of the “current value” of the externaldevice in the monitoring apparatus 1. The display screen 41 is displayedby the CPU that serves as the computer inside of the control unit 21 ofthe client apparatus 2 executing a program that is stored on the storageunit 22 of the client apparatus 2 based on instructions for performingthe setting of a superimposed display by the user.

An image capturing screen is displayed on the entire display screen 41.In addition, the external devices 3 a, and 3 b, an add button 411 foradding a new superimposed display, a delete button 412 for deleting asuperimposed display, a subject 413, and superimposed display settingboxes 414, and 415 that display the current values for the externaldevices 3 a, and 3 b are further displayed on the display screen 41.

A superimposed display in which each current value for the externaldevices that have already been added on the display 31 by the user isdisplayed on the display screen 41. Thereby, upon the add button 411being clicked, a display box such as, for example, the superimposeddisplay setting box 414, is displayed on the display screen 41. Next, byindicating the device number for the external device that is associatedwith the superimposed display setting box 414, the current value thathas been acquired from the associated external device is displayedinside of the superimposed display setting box 414.

In the superimposed display setting box 414, for example, 80%, which isthe current value acquired from the external device 3 a, is displayed.Note that this can be made a configuration that takes into account thatthe external device holds multiple types of functions, and indicates thetype of function for the current value that is being displayed inaddition to the device number, or each of the current values for theplurality of types of functions may also be displayed.

In addition, the display is not limited to the “current value” for thefunctions of the external device, and other values such as for example,the “drive state” may also be displayed. In addition, settings thatindicate the scope of the angle of view for the pan and tilt, and thezoom ratio that display the superimposed display may also be carriedout.

Next, the user uses the operating unit 24 such as the mouse and the likeof the client apparatus 2 and disposes the superimposed display settingbox 414 in the vicinity of the corresponding external device 3 a on thedisplay screen 41 by dragging and dropping the superimposed displaysetting box 414. Specifically, this is disposed in the vicinity of thecorresponding external device (for the superimposed display setting box415, this is the external device 3 b) similar to the superimposeddisplay setting box 415.

The information that has been obtained in this manner such as thesetting state for the superimposed display setting box, and the like isstored on the storage unit 12 of the monitoring apparatus 1. The controlunit 11 generates an image (a video image) in which a superimposeddisplay has been superimposed on a captured image based on theinformation for the superimposed display setting box that has beenstored, and stores this on the storage unit 12 or the storage unit 22 toserve as a superimposed image (superimposed video image).

In addition, superimposed display setting boxes can be deleted byselecting the superimposed display setting box that the user would liketo delete on the screen using the mouse or the like and clicking thedelete button 412.

FIG. 5 is a diagram showing one example of a display screen 51 on whicha superimposed image according to the First Embodiment has beendisplayed. The display screen 51 is displayed by the program that hasbeen stored on the storage unit of the client apparatus 2 being executedaccording to an instruction to perform display of the display screenbeing output by the client apparatus 2.

The display screen 51 of FIG. 5 displays the external devices 3 a, and 3b, the subject 413, and superimposed displays 511, and 512 ofinformation relating to the external device (the brightness of theilluminating unit, or the like) corresponding to each of the externaldevices 3 a, and 3 b that have been set on the display screen 41.

Next, an example of the operations of the monitoring apparatus 1 will beexplained in reference to the flowchart in FIG. 6 . FIG. 6 is aflowchart showing one example of processing for the monitoring apparatusaccording to the First Embodiment, and the flowchart in FIG. 6 isrealized by the CPU that is provided inside of the control unit 11 ofthe monitoring apparatus 1 executing a program that has been stored onthe storage unit 12.

During step S61, the control unit 11 receives an indicator flashinstruction command from the client apparatus 2 via the communicationsunit 15. At this time, the user clicks the indicator flash buttons 315,and 316 in the management information for the external device that theywhich to specify among the external devices that are displayed on thedisplay screen 31.

Upon these buttons being clicked, an indicator flash instruction commandis transmitted by the client apparatus 2, which is received by thecommunications unit 15 of the monitoring apparatus 1, and stored on thestorage unit 12. The example in the present flowchart assumes that theuser has clicked, for example, the indicator flash button 315 of themanagement information 311 in order to specify the external device 3 a.

During step S62, the control unit 11 determines whether or not theexternal device that was the target of the indicator flash instructionis an electric light type device. Specifically, the control unit 11acquires the information for the external device for the indictor flashcommand from the storage unit 12, and determines whether or not this isan electric light type device.

At this time, in a case in which the communication standard for thenetwork 5 is for example, Z-wave, in a case in which it has beendetermined that the Device Type included in this information is LightDimmer Switch or Color Switch, it is determined to be an electric lighttype device. Alternatively, in a case in which this external devicesupports a Color Switch command class, it is determined to be anelectric light type device.

In addition, the processing may also be made such that a file that is adatabase in which the device type for each type of external device hasbeen recorded is stored on the storage unit 12 in advance, and whetheror not the target external device is an electric light type device isreferenced in this database file.

In a case in which it has been determined that the device is an electriclight type device, the processing proceeds to step S64, and if this isnot the case, the process proceeds to step S63. Note that in the examplein FIG. 6 , the user clicks the indicator flash button 315 in themanagement information 311 in order to specify the external device 3 a,and therefore, the processing proceeds to step S64.

During step S63, the control unit 11 transmits an indicator flashinstruction command to the external device that had an indicator flashinstruction during step S61 via the communications unit 15, and theprocessing in the present flowchart is completed. For example, in a casein which the communication standard for the network 5 is Z-Wave, theindicator flash instruction command uses an indicator set command or thelike. The indicator of the external device 3 a, which is not indicated,thereby flashes.

During step S64, the control unit 11 determines whether or not theelectric light (illuminating unit) of the target external device iscurrently lit up. Specifically, the control unit 11 transmits a commandto confirm the current value for the present electric light to thetarget external processing device, and performs the determinationaccording to the contents of the response thereto. At this time, thecontrol unit 11 determines that the illuminating unit is in use in acase in which the illuminating unit is lit up.

In the present embodiment, it is assumed that the communicationsstandard for the network 5 is, foe example, Z-Wave. Therefore, theCurrent Value or Target Value for a Multi-level Switch Report, binaryswitch report, or color switch report that are included in theinformation acquired from this external device are confirmed. If thesevalues are 0x01 or greater, it is determined that the electric light iscurrently lit up.

In addition, the processing may also be made such that the value forwhen a light up command was made for an external device that waspreviously a target is stored to serve as the current value for theelectric light in the storage unit 12, and whether or not the electriclight is currently lit up is determined based on this current value. Ina case in which it has been determined that the electric light iscurrently lit up, the processing proceeds to step S65, and if this isnot the case, the processing proceeds to step S63.

During Step S65, the control unit 11 determines whether or not todisplay a mark, or the like for specifying an external device on thescreen of the display unit of the client apparatus 2. The control unit11 therefore transmits a command to display the display screen 71 ofFIG. 7 to the client apparatus 2 via the communications unit 15, and thedisplay screen 71 is displayed on the display unit 6 of the clientapparatus 2.

That is, step S65 functions as a control step that controls a displayscreen that performs superimposed display of information relating to anexternal device on a video image including the external device that wascaptured by an image capturing unit. In addition, during step S65, ascreen is displayed for displaying predetermined specifying informationin the vicinity of the external device on the display screen. In thiscontext, the screen for displaying the specifying information includes ascreen that prompts the display of specifying information (displayscreen 71 of FIG. 7 ).

FIG. 7 is a diagram that shows one example of a display screen forprompting the display of specifying information on the display unit ofthe client apparatus 2 according to the First Embodiment. The displayscreen 71 that is shown in FIG. 7 is displayed by the CPU that isprovided inside of the control unit of the client apparatus 2 executinga program that is stored on the storage unit of the client apparatus 2in response to the command that has been sent from the control unit 11during step S65.

If the indicator of the external device that has been selected flashes,this will influence the lighting environment, and therefore, a dialoguefor confirming if specifying information (a mark or the like) forspecifying the external device should be displayed on the viewer screeninstead is displayed on the display screen 71.

In addition, a button 711 for determining to display the specifyinginformation (mark or the like) for specifying the external device on theviewer screen and a button 712 for determining to not display thespecifying information for specifying the external device on the viewerscreen are displayed on the display screen 71.

Note that the control unit 11 may also confirm whether or not thedisplay of the specifying information (mark or the like) for specifyingthe target external device has been set from specifying informationdisplay settings that are stored on the storage unit 12. In addition, ina case in which specifying information display settings have not beenstored, the control unit 11 may also instruct the client apparatus 2 soas to display a warning asking “The electric light flashing may affectthe lighting environment. Would you still like to make it flash?”. Thatis, the screen prompting the display of specifying information alsoincludes, for example, screens that display warnings and the like.

By the display screen 71 being displayed and the user clicking eitherthe button 711 or the button 712, determination information istransmitted to the monitoring apparatus 1 by the communications unit ofthe client apparatus 2, and the determination information that has beenreceived via the communications unit 15 is stored on the storage unit12.

During step S65, whether or not the determination information that hasbeen received is determination information for displaying the specifyinginformation on the viewer screen is determined. In a case in which thebutton 712 has been clicked and No has been determined in step S65 (acase in which the display of the specifying information has not beenset), the present flowchart is completed after making the indicatorflash during step S63.

That is, if the illuminating unit of the external device is not beingused, the illuminating unit is made to light up or flash during stepS63. Note that if the button 711 from FIG. 7 has been clicked on theclient apparatus side, the display may also transition immediately tothe display screen 81 on the client apparatus side.

In a case in which Yes has been determined during step S65, theprocessing proceeds to step S66, and the control unit 11 determineswhether or not a superimposed display of information (the brightness ofthe illuminating unit, or the like) relating to the target externaldevice is included in the angle of view that is captured by the imagecapturing unit 13.

By the image capturing control unit 16 changing the angle of view, it ispossible that the superimposed display of the target external devicethat was set on the display screen 41 will be made to be outside of theangle of view. That is, there are cases in which the superimposeddisplay of the external device is not included in the angle of view (inthe screen) due to the pan/tilt angle or the zoom ratio at which imagesare being captured by the monitoring apparatus 1.

Therefore, during step S66, the control unit 11 determines if all of thecoordinates for the superimposed display of information relating to thetarget external device that are stored on the storage unit 12 areincluded in the current angle of view of the captured image.

In addition, in cases in which even when the superimposed display isinside of the angle of view of the captured image, but settings havebeen performed to limit the range of the angle of view at which thesuperimposed display is actually being displayed, it is determined thatthe superimposed display of information relating to the target externaldevice is included in the angle of view only when the angle of view iswithin that range. This is because there are cases in which settingshave been performed to display the superimposed display only in aspecific range of pan/tilt angles and zoom ratios, or the like.

During step S66, in a case in which it has been determined that thesuperimposed display of the information relating to the target externaldevice is included in the angle of view, the processing proceeds to stepS68, and if this is not the case, the processing proceeds to step S67.Note that in a case in which the monitoring apparatus 1 does not have afunction to change the angle of view, the processing also proceeds tostep S68.

During step S67, the control unit 11 changes the angle of view such thatthe superimposed display is displayed within the angle of view of thecaptured image. Specifically, the control unit 11 performs aninstruction for changing the angle of view to the image capturingcontrol unit 16 such that all of the coordinates for the superimposeddisplay of the information relating to the target external device thatare stored on the storage unit 12 are inside of the angle of view forthe superimposed display. That is, it changes the angle of view suchthat the superimposed display of the information relating to theexternal device is displayed inside of a predetermined angle of view ofthe video image.

The change instruction for the angle of view can use any kind of methodas long as it is a change instruction such that all of the coordinatesfor the superimposed display are included in the angle of view for thesuperimposed display. For example, a change instruction may be made suchthat the angle of view from when the superimposed display has been seton the display screen 41 is stored on the storage unit 12, and the angleof view is made the same as this stored angle of view, or a changeinstruction may also be made such that the superimposed display isdisplayed on the center of the screen.

During step S68, the control unit 11 superimposes and displays thespecifying information for the target external device on the capturedimage as is shown in FIG. 8 . FIG. 8 is a diagram showing one example ofa display screen on which specifying information has been superimposedon a viewer according to the First Embodiment.

As is shown in FIG. 8 , the control unit 11 generates a superimposedimage in which specifying information 811 (an icon or mark or the like)for specifying the target external device has been superimposed on thecaptured image. That is, a screen (the display screen 81 in FIG. 8 ) fordisplaying predetermined specifying information in the vicinity of theexternal device is displayed.

The video image stream that is based on the generated superimposed imageis streamed to the client apparatus 2 via the communications unit 15,and a viewer screen such as the display screen 81 in FIG. 8 is displayedon the display unit 6 of the client apparatus 2.

That is, the captured image is displayed on the entirety of the displayscreen 81, and the external devices 3 a, and 3 b, the subject 413, asuperimposed display 512 corresponding to the current value of theexternal device 3 b that was set on the display screen 41, andspecifying information 811 (an icon or a mark or the like) specifyingthe external device 3 a are further displayed.

Note that when generating the superimposed image, the control unit 11superimposes specifying information 811 such as an icon, a mark, or thelike that specifies the external device 3 a instead of the current valueor the like of the external device 3 a, or as additional information, inthe position of the superimposed display 511 in FIG. 5 . In addition,the generated superimposed image (superimposed video image) is stored onthe storage unit 12. Note that, the specifying information 811 may alsobe an icon, mark, or the like showing a shape such as for example, astar, a circle, or a square, or it may also be an icon, mark, or thelike with any kind of shape or color.

Alternatively, the information relating to the external device thatserves as the specifying information (the device number, the name andcurrent value of this device, or the like) may also be displayed so asto stand out. That is, the superimposed specifying information (thedevice number, the name, and the current value, or the like) may also bemade information that is made to periodically flash, for which the sizehas been changed, or for which the color or brightness has been changed,or an emphasized display such that this stands out even more may also beperformed.

That is, the information relating to the external device that serves asthe specifying information may also include at least one of the devicenumber for the external device, the name of the external device, or thecurrent value relating to the operational state of the external device.

In addition, this may also be changed to a way of emphasizing thedisplay according to the drive state of the external device such aschanging the color to green when the external device is on, yellow whenit is in sleep mode, red when it is off, or the like. In addition, in acase in which the average luminance around the specifying information ishigh or low, it may also be changed such that the color of thespecifying information is changed to be darker or brighter, or the like.

Note that the present embodiment has explained an example in whichspecifying information is displayed as being superimposed on a capturedimage that is currently being captured by the image capturing unit 13.However, for example, snapshot images from when superimposed displaysettings for the target external device were performed on the displayscreen 41 may be stored on the storage unit 12, and these snap imagesmay be used for the superimposed display of the specifying information.

Specifically, in a case in which the control unit 11 has determined thatspecifying an external device on the viewer screen has been determinedduring step S65, the processing proceeds to step S68. In addition,during step S68, when the superimposed image in which the specifyinginformation is superimposed is being generated, the specifyinginformation may also be superimposed in the position of the superimposeddisplay settings for the target external device on the previouslydescribed snapshot image.

As has been described above, in the First Embodiment, when the userclicks on the flash indicator button on the display screen 31 for theexternal device to be specified, in a case in which the electric lightfor the target external device is lit up, information (a mark or thelike) specifying the target external device is displayed as beingsuperimposed on the image.

That is, during the image processing method that is shown in step S61 tostep S68, when an instruction to light up or flash the illuminating unitof an external device has been input, the control unit 11 determineswhether or not the illuminating unit for this external device is in use.Then, in a case in which it is in use, instead of lighting up or makingthe illuminating unit flash, a screen is displayed on the display screenin order to display predetermined specifying information in the vicinityof the external device.

Therefore, even in cases in which monitoring is being performed whileilluminating a subject using the electric lamp that serves as theilluminating unit, it is possible to specify and confirm the externaldevice on the viewer screen without making the electric light flash.Note that in cases in which even if the indicator is made to flashinstead of the electric light, in cases in which the strength of lightemission for the indicator is strong, there are cases in which this willcause a negative influence, and therefore, by specifying and confirmingan external device on the viewer screen, it is possible to suppress thisnegative influence.

Second Embodiment

Although in the First Embodiment, a case in which whether or not theelectric light of the external device is lit up is determined wasexplained, in the Second Embodiment, whether or not image detection hasbeen set will be explained. Note that the configuration of themonitoring system according to the present embodiment is the same asthat in the First Embodiment.

FIG. 9 is a diagram showing one example of a display screen 91 forsetting image detection of an external device in the monitoringapparatus 1 according to the Second Embodiment. The display screen 91 isdisplayed by the CPU inside of the control unit of the client apparatus2 executing a program that is stored on the storage unit of the clientapparatus 2 based on an instruction to perform image detection setting.

The external devices 3 a, and 3 b, an add button 91 for adding a newimage detection region, a delete button 912 for deleting an imagedetection region that has been added, an image detection region 913, andthe superimposed displays 511, and 512 that display the current valuesfor the external devices 3 a, and 3 b, and the like, are displayed onthe display screen 91.

The user performs settings in advance for image detection on the videoimage that is being captured by the image capturing unit 13. The videodetection may be any kind of video image detection such as, for example,moving body detection that detects a moving body in the detectionregion, as long as it analyses and performs detection on a video image.When the user clicks the add button 911, a setting box for a detectionregion such as the image detection region 913 is displayed.

After this, an input device such as a mouse or the like that has beenconnected to the client apparatus 2 is used, and the image detectionregion is moved to an appropriate position within the screen, or thesize of the image detection region is changed by dragging and droppingthe setting box. In addition, in a case in which the user would like todelete the setting box, they can delete it by clicking the delete button912 after having clicked on this setting box.

Note that the image detection may also be set in any way according tothe type of image detection, such as setting the image detection to theentire screen, or drawing a border line that determines the detectionborder. The image detection settings that are obtained in this mannerare stored on the storage unit 12 via the communications unit 15 of themonitoring apparatus 1 by the communications unit of the clientapparatus 2.

Next, an example of the processing for the case in which image detectionhas been set will be explained in reference to a portion of theflowchart in FIG. 10 . FIG. 10 is a flowchart showing one example ofprocessing for a monitoring apparatus according to the SecondEmbodiment. Note that the operations for each step of the flowchart inFIG. 10 are performed by the CPU that serves as the computer inside ofthe control unit 11 executing a computer program that has been stored ona memory.

In the flowchart in FIG. 10 , the processing for the steps that havebeen given the same reference numbers as the steps in flowchart in FIG.6 is the same as the processing for these steps, and thereforeexplanations thereof will be omitted. In FIG. 10 , instead of processingfor determining whether or not an electric light is lit up, during stepS101, the control unit 11 determines whether or not image detection hasbeen set.

Specifically, the control unit 11 acquires whether or not imagedetection settings were performed on the display screen 91 from thestorage unit 12. In a case in which it has been determined that imagedetection has been set, the processing proceeds to step 65, and if thisis not the case, the process proceeds to step S63.

Note that further additional conditions may also be set, not just imagedetection being set. For example, during step S101, the control unit 11references whether or not image detection has been set, and whether ornot rules have been set is determined by referring to the storage unit12 to perform predetermined operations with an image detection eventserving as a trigger.

In addition, if the result of step S101 is Yes, the processing proceedsto step S65, and in a case on No, the processing proceeds to step S63.That is, if an image recognition operation based on the video image isnot in progress, the illuminating unit is made to light up or flashduring step S63.

In addition, in a case in which an image detection event is occurring,there are cases in which, for example, it is not desirable for changesto the lighting environment to occur because some sort of phenomena suchas a moving body passing by, or the like is currently occurring. Thus,the processing may also be made such that the control unit 11 referencesthe storage unit 12 about whether an image detection event is occurring,and in a case in which one is occurring, the processing proceeds to stepS65, and in a case in which one is not, the processing proceeds to stepS63.

In this manner, in the Second Embodiment, when the user has clicked theindicator flash button on the display screen 31 for the external devicethat they would like to specify, in a case in which image detection hasbeen set, information (a mark, or the like) that specifies the targetexternal device is displayed as superimposed on the image.

That is, in the Second Embodiment, in step S61 to step S68 of FIG. 10 ,when a light up or flash instruction for the illuminating unit of theexternal device has been input, the control unit 11 determines whetheror not an image recognition operation based on the video image is inprogress. In addition, in a case in which an image recognition operationis in progress, a screen for displaying predetermined specifyinginformation in the vicinity of the external device is displayed on thedisplay screen instead of making the illuminating unit light up orflash.

Therefore, it is possible to specify the external device on the viewerscreen without making the indicator flash, and it is possible to preventerroneous operations for the image detection operations caused bychanges occurring to the lighting environment due to the indicatorflashing.

Other Embodiments

Above, the present invention has been described based on preferredembodiments (examples) thereof. However, the present invention is notlimited to the above-described embodiments (examples), and variouschanges based on the gist of the present invention are possible, andthese changes are not excluded from the scope of the present invention.

For example, a portion or the entirety of the functional configurationsof the above-described monitoring system may also be made hardware, andmay be included in the monitoring apparatus 1, the client apparatus 2,and the external devices 3 a, and 3 b.

In addition, the configurations, processing, and the like of eachembodiment that was described above may also be applied to a deviceother than the monitoring apparatus 1, the client apparatus 2, and theexternal devices 3 a, and 3 b, such as a computer, another type ofdevice, or the like. That is, the processing and operations that weredescribed in FIG. 3 to FIG. 10 may also be performed by the control unit11 of the monitoring apparatus 1, or they may also be performed by theclient apparatus 2.

Furthermore, they may also be performed by devices other than these, orthe monitoring apparatus 1, the client apparatus 2, and the devicesother than these may perform them together. Therefore, the imageprocessing apparatus of the present embodiment includes the monitoringapparatus 1, the client apparatus 2, and devices other than these.

In addition, in the explanation, the external devices that configure themonitoring system of the above-described embodiments were made theexternal devices 3 a, and 3 b. However, the number of external devicesdoes not matter as long as there is one or more.

In addition, at least one portion of the configuration of the monitoringapparatus 1 may also be realized using hardware. In a case in which thisis realized by hardware, it is sufficient if, for example, dedicatedcircuits are automatically generated from a program for realizing eachstep on an FPGA (field programmable gate array) by using a predeterminedcompiler.

In addition, the processing may also be made such that gate arraycircuits are formed in the same manner as the FPGA, and are realized ashardware. In addition, they may also be realized by an ASIC (ApplicationSpecific Integrated Circuit).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation toencompass all such modifications and equivalent structures andfunctions.

In addition, as a part or the whole of the control according to theembodiments, a computer program realizing the function of theembodiments described above may be supplied to the image processingapparatus through a network or various storage media. Then, a computer(or a CPU, an MPU, or the like) of the image processing apparatus may beconfigured to read and execute the program. In such a case, the programand the storage medium storing the program configure the presentinvention.

This application claims the benefit of Japanese Patent Application No.2022-090359, filed on Jun. 2, 2022, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising at leastone processor or circuit configured to function as: a control unitconfigured to control a display screen that performs superimposeddisplay of information relating to an external device on a video imageincluding the external device that has been captured by an imagecapturing unit; and determining unit configured to determine whether theilluminating unit of the external device is in use or not, or whether animage recognition operation based on the video image is in progress ornot; wherein when an instruction to make an illuminating unit of theexternal device light up or flash has been input, in a case in whichdetermining unit determines that the illuminating unit of the externaldevice is in use, or an image recognition operation based on the videoimage is in progress, the control unit displays a screen for displayingpredetermined specifying information in the vicinity of the externaldevice on the display screen instead of making the illuminating unitlight up or flash.
 2. The image processing unit according to claim 1,wherein, in a case in which the illuminating unit is lit up, the controlunit determines that the illuminating unit is in use.
 3. The imageprocessing unit according to claim 1, wherein the specifying informationincludes information in which the color or brightness of the informationrelating to the external device has been changed.
 4. The imageprocessing apparatus according to claim 3, wherein the informationrelating to the external device includes at least one of a device numberfor the external device, a name of the external device, and a currentvalue relating to an operational state of the external device.
 5. Theimage processing apparatus according to claim 1, wherein the specifyinginformation includes a predetermined icon or mark.
 6. The imageprocessing apparatus according to claim 1, wherein the control unitchanges the angle of view such that the specifying information isdisplayed within a predetermined angle of view of the video image. 7.The image processing apparatus according to claim 1, wherein the screenfor displaying the specifying information includes a screen that promptsthe display of the specifying information.
 8. The image processingapparatus according to claim 7, wherein the screen that prompts thedisplay of the specifying information includes a screen that displays awarning.
 9. The image processing apparatus according to claim 1, whereinwhen the instruction to make the illuminating unit of the externaldevice light up or flash has been input, if the illuminating unit of theexternal device is not in use, or an image recognition operation basedon the video image is not in progress, the control unit makes theilluminating unit light up or flash.
 10. The image processing apparatusaccording to claim 1, wherein the illuminating unit includes anindicator.
 11. The image processing unit according to claim 1, whereinthe illuminating unit includes an electric light.
 12. An imageprocessing method comprising: a control step for controlling a displayscreen that performs superimposed display of information relating to anexternal device on a video image including the external device that hasbeen captured by an image capturing unit; and determining step fordetermining whether the illuminating unit of the external device is inuse or not, or whether an image recognition operation based on the videoimage is in progress or not; wherein when an instruction to make anilluminating unit of the external device light up or flash has beeninput, in a case in which determining step determines that theilluminating unit of the external device is in use, or an imagerecognition operation based on the video image is in progress, thecontrol step displays a screen for displaying predetermined specifyinginformation in the vicinity of the external device on the display screeninstead of making the illuminating unit light up or flash.
 13. Anon-transitory computer-readable storage medium configured to store acomputer program comprising instructions for executing followingprocesses: a control step for controlling a display screen that performssuperimposed display of information relating to an external device on avideo image including the external device that has been captured by animage capturing unit; and determining step for determining whether theilluminating unit of the external device is in use or not, or whether animage recognition operation based on the video image is in progress ornot; wherein when an instruction to make an illuminating unit of theexternal device light up or flash has been input, in a case in whichdetermining step determines that the illuminating unit of the externaldevice is in use, or an image recognition operation based on the videoimage is in progress, the control step displays a screen for displayingpredetermined specifying information in the vicinity of the externaldevice on the display screen instead of making the illuminating unitlight up or flash.